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PREFACE 


As  a  private  nonprofit  corporation.  New  American  Schools  (NAS)  be¬ 
gan  in  1991  to  fund  the  development  of  designs  aimed  at  transform¬ 
ing  entire  schools  at  the  elementary  and  secondary  levels.  Having 
completed  its  competition,  development,  and  demonstration 
phases,  NAS  currently  is  scaling-up  its  designs  to  form  a  critical  mass 
of  schools  within  several  partnering  districts. 

The  purpose  of  this  report  is  to  describe  the  baseline  characteristics, 
such  as  school  demographics  and  performance,  of  a  large  number 
of  NAS  sites  in  the  early  implementation  stages  of  NAS’s  scale-up 
phase.  During  this  phase,  RAND’s  research  activities  include  moni¬ 
toring  the  progress  of  a  large  sample  of  NAS  schools  in  eight  jurisdic¬ 
tions  from  the  1994-1995  school  year  through  the  1999-2000  school 
year.  This  report  provides  a  description  of  the  schools’  demograph¬ 
ics,  climate,  and  test  scores.  The  next  step  in  the  RAND  analysis  is  to 
report  on  trends  in  implementation,  school  performance,  and  re¬ 
lated  factors. 

RAND  will  monitor  changes  in  this  sample  of  schools  over  the  next 
several  years.  This  report  establishes  a  baseline  for  tracking  the 
changes.  The  hope  is  that  all  those  interested  in  school  improve¬ 
ment-parents,  teachers,  administrators,  policymakers,  researchers, 
community  members,  etc. — ^will  benefit  from  the  analysis. 

Other  RAND  reports  about  New  American  Schools  include 

•  Lessons  from  New  American  Schools’ Scale-Up  Phase:  Prospects 
for  Bringing  Designs  to  Multiple  Schools,  Susan  J.  Bodilly,  1998 
(MR-1777-NAS). 
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SUMMARY 


Upon  its  founding  in  1991,  New  American  Schools  (NAS)  sought  to 
engage  the  nation’s  best  educators,  business  people,  and  researchers 
in  the  task  of  creating,  testing,  and  fostering  the  implementation  of 
schoolwide  designs  not  constrained  by  existing  regulations,  rules, 
and  conventions.  These  beginnings  of  “break  the  mold”  schools 
evolved  to  the  point  where,  in  1995,  NAS  began  to  widely  diffuse  the 
designs  to  a  large  number  of  schools  within  partnering  school  dis¬ 
tricts — a  period  known  as  its  scale-up  phase.  NAS's  mission  during 
this  phase  has  been  to  use  design-based  assistance  to  help  schools 
and  districts  raise  the  achievement  of  large  numbers  of  students. 

Initially,  RAND  helped  NAS  design  a  request  for  proposals  and  con¬ 
duct  a  national  competition  for  designs.  Subsequently,  RAND  stud¬ 
ied  the  initial  implementation  of  the  designs  during  the  development 
and  demonstration  phases,  providing  feedback  to  both  NAS  and  the 
design  teams.  With  the  beginning  of  scale-up  of  the  NAS  designs  in 
the  fall  of  1995,  RAND’s  research  agenda  expanded  to  provide  a 
comprehensive  assessment  that  aims  to  inform  NAS,  the  design 
teams,  partnering  jurisdictions,  educators,  and  the  general  public 
with  lessons  learned  about  comprehensive  school  reform 

RAND’S  ASSESSMENT  OF  NEW  AMERICAN  SCHOOLS 

NAS’s  approach  is  ambitious  and  complex.  The  strategy  relies  on 
school  designs  and  design-based  assistance  to  change  the  school  or¬ 
ganization,  the  professional  life  of  teachers,  and  classroom  environ¬ 
ments  as  a  way  to  improve  student  learning  opportunities  and 
achievement.  A  variety  of  institutional,  social,  political,  and  eco- 
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nomic  factors  need  to  be  aligned  for  successful  implementation, 
consistency  and  coherence,  and  ultimate  success.  As  a  result,  any 
assessment  of  the  NAS  initiative  has  to  be  multidimensional  and 
multifaceted  and  must  rely  on  a  variety  of  data  and  a  wide  array  of 
indicators.  It  is  a  difficult  and  analytically  complex  undertaking.  Yet 
it  is  critical  to  understanding  the  progress  of  schools  implementing 
schoolwide  reform,  especially  because  so  little  research  has  been 
conducted  in  this  relatively  new  educational  policy  area. 

Critical  Issues 

The  NAS  design  teams  and  the  assistance  they  provide  aim  to  pro¬ 
mote  greater  cohesiveness  and  coherence  in  a  school’s  mission  and 
programs  as  well  as  collaboration  among  its  staff.  One  hypothesis  is 
that  over  time,  implementation  within  each  school  will  deepen  and 
all  school  staff  will  embrace  and  engage  in  activities  supportive  of  the 
visions  of  the  design,  especially  if  the  design  continues  to  provide 
assistance  and  teachers  continue  to  become  more  familiar  with  it. 
Each  design  may  then  become  schoolwide,  and  a  second  hypothesis 
can  be  examined:  schoolwide  designs  will  improve  teachers’  and 
students’  educational  opportunities  and  performance. 

However,  a  danger  in  educational  reform  initiatives — especially 
those  within  urban  settings  with  complex  economic,  political,  and 
social  challenges — is  that  designs  may  be  just  another  "program” 
that  is  turned  on  and  off  at  selected  times  during  the  school  day, 
week,  and/or  year.  As  time  goes  on,  the  NAS  designs  may  be  at  risk 
of  being  turned  off  altogether,  especially  if  districts  and  schools  lose 
their  focus  on  NAS  and  turn  to  other  reform  efforts. 

This  issue  is  one  that  RAND  will  address  with  its  longitudinal  sample 
of  implementing  NAS  schools.  If  teachers,  design  teams,  and  dis¬ 
tricts  can  sustain  their  focus  on  the  NAS  designs  to  structure  the  edu¬ 
cational  opportunities  of  students  and  teachers,  the  designs  will 
likely  become  more  widespread  within  schools. 

In  future  analyses  we  will  examine  trends  in  implementation  to  un¬ 
derstand  the  degree  to  which  key  design  team  features  become  inte¬ 
gral  components  of  the  schools  in  our  sample.  In  addition,  we  will 
report  trends  in  school  performance  indicators  as  provided  by  dis¬ 
tricts  in  their  public  reports  about  the  schools’  test  score  perfor- 
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mance,  absenteeism  rates,  and  disciplinary  problems.  Most  impor¬ 
tant,  we  will  seek  explanations  for  school  performance  trends  within 
the  participating  school  districts.  Such  explanations  are  critical  for 
drawing  lessons  as  the  nation  moves  farther  along  the  path  of  com¬ 
prehensive  school  reform  and  considers  it  as  a  productive  way  to 
spend  federal  and  local  funds. 

Key  Questions 

To  assess  this  complex  effort,  we  are  conducting  analyses  aimed  at 
answering  five  key  questions: 

•  What  were  the  NAS  schools  like  before  they  implemented  the 
designs? 

•  How  have  the  designs  and  the  assistance  they  provide  evolved 
over  time? 

•  Are  the  critical  components  of  the  NAS  designs  being  imple¬ 
mented  across  a  wide  array  of  schools?  Why  or  why  not? 

•  Do  the  NAS  designs  extend  beyond  changes  in  school  organiza¬ 
tion  and  governance  and  permeate  classrooms  to  change  cur¬ 
riculum  and  instruction? 

•  Over  time,  what  is  the  progress  of  the  schools  being  assisted  by 
NAS  design  teams  in  improving  student  and  school  perfor¬ 
mance? 

Data  Collection 

RAND  is  using  a  variety  of  data  to  address  the  key  questions.  Along 
with  the  district-provided  data,  we  have  collected  information 
through  interviews  with  principals  and  teachers,  focus  groups,  case 
studies,  surveys,  observations,  school  and  classroom  artifacts,  and 
document  analysis.  In  a  supplementary  study  of  classroom  prac¬ 
tices,  we  are  administering  a  commercial  test  to  elementary  students 
in  NAS  and  non-NAS  schools  in  San  Antonio,  Texas.  All  of  these  data 
inform  our  understanding  of  the  reform  that  NAS  is  attempting  to 
accomplish  in  districts,  schools,  classrooms,  and  the  academic  life  of 
the  nation’s  students. 
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PURPOSE  OF  THIS  REPORT 

This  report  has  a  dual  purpose.  First,  it  presents  our  data  collection 
plan  for  addressing  the  key  questions.  The  plan  is  discussed  within 
the  context  of  the  overall  NAS  effort.  Second,  it  presents  our  findings 
from  answering  the  first  question;  What  were  the  NAS  schools  like 
before  they  implemented  the  designs?  We  rely  on  data  from  partner¬ 
ing  school  districts  and  retrospective  accounts  provided  by  princi¬ 
pals  during  phone  interviews  conducted  in  the  spring  of  1997.  These 
data  provide  information  about  schools’  demographic,  climate,  and 
performance  characteristics  at  the  beginning  of  NAS’s  scale-up 
phase. 

SUMMARY  OF  FINDINGS  AND  FUTURE  WORK 

For  the  most  part,  NAS  design  teams  began  assisting  schools  facing 
many  academic  and  social  challenges.  The  majority  of  the  NAS 
schools  are  urban,  low-achieving,  poor,  and  predominantly  minority. 
Baseline  information  from  the  1994-1995  school  year  reveals  that 
NAS  schools,  on  average,  were  performing  at  or  below  the  district  av¬ 
erage  on  the  mandated  tests  the  year  before  designs  began  assisting 
schools  in  their  restructuring  efforts.  Generally,  then,  NAS  designs 
are  attempting  to  change  the  academic  lives  of  students  and  the 
professional  lives  of  teachers  in  difficult  environments. 

NAS  design  teams  peirtnered  with  schools  that  had  challenging  cli¬ 
mates.  We  compared  the  NAS  school  climate  indicators  to  a  national 
sample.  A  greater  percentage  of  NAS  principals  reported  problems — 
including  poverty,  difficulties  with  the  English  language,  racial- 
ethnic  tension,  lack  of  school  readiness,  student  and  teacher  absen¬ 
teeism,  and  verbal  abuse  and  disrespect  of  teachers — than  did  prin¬ 
cipals  in  the  national  sample. 

Whether  design  teams  and  the  assistance  they  provide  to  these 
schools  improve  students’  educational  opportunities  and  perfor¬ 
mance  is  an  open  question.  Future  RAND  reports  will  document  and 
explain  trends  in  both  implementation  and  school  performance,  and 
the  social  and  organizationcd  factors  related  to  them.  The  aim  of 
our  analysis  is  to  monitor  the  implementation  and  performance 
trends  in  a  large  sample  of  NAS  schools  and  to  explain  why  those 
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trends  occur.  With  such  knowledge,  we  will  provide  some  additional 
lessons  learned  about  the  conditions  under  which  designs  can  work 
effectively  to  improve  student  achievement  across  the  nation. 
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_ _ Chapter  One 

INTRODUCTION  AND  RESEARCH  QUESTIONS 


For  some  time,  there  have  been  many  debates  about  the  need  to  re¬ 
organize  schools  to  create  learning  opportunities  more  consistent 
with  the  changing  needs  of  American  society.  Concerns  about  how 
well  the  American  education  system  is  serving  societal  needs  have  a 
long  history  in  this  nation  (Kliebard,  1986;  Cuban,  1990;  Tyack  and 
Cuban,  1995).  For  example,  the  1983  National  Commission  on  Excel¬ 
lence  in  Education’s  report,  A  Nation  at  Risk,  stated,  “Our  once  un¬ 
challenged  preeminence  in  commerce,  industry,  science,  and  tech¬ 
nological  innovation  is  being  taken  over  by  competitors  throughout 
the  world. ...  If  an  unfriendly  power  had  attempted  to  impose  on 
America  the  mediocre  educational  performance  that  exists  today,  we 
might  have  viewed  it  as  an  act  of  war.”  Not  surprisingly,  new  educa¬ 
tional  interventions  and  experiments  emerged. 

The  debates  about  the  state  of  education  in  the  United  States  will 
continue.  And  given  that  the  recent  release  of  the  Third  Interna¬ 
tional  Mathematics  and  Science  Study  (TIMSS)  showed  that  U.S. 
high  school  students  continue  to  lag  behind  those  of  other  countries, 
efforts  to  improve  student  achievement  scores  in  this  country  will 
also  continue.  In  addition,  as  educational  and  income  inequalities 
persist  and  American  society  becomes  increasingly  diverse,  attention 
to  equity  will  continue  as  well  (see  Kozol,  1991;  Jencks  and  Peterson, 
1991;  Smith,  1995;  National  Research  Council,  1997;  Jencks  and 
Phillips,  1998). 

The  emerging  reports  on  the  condition  of  American  education  and 
new  policies  and  innovative  activities  have  established  a  positive 
climate  for  reform.  The  conventional  wisdom  is  that  schools  should 
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and  could  do  a  better  job  of  helping  students  learn  basic  and  higher- 
order  skills  in  reading,  writing,  mathematics,  science,  and  technology 
(Newmann  and  Associates,  1996;  Johnson  and  Immerwahr,  1994). 

THE  NEW  AMERICAN  SCHOOLS  REFORM  EFFORT 

In  this  context,  with  attention  directed  toward  improving  equity  and 
excellence,  the  New  American  Schools  (NAS)  initiative  was  launched 
during  the  summer  of  1991.  National  educational  goals  emerged 
from  policy  forums  such  as  the  1989  meeting  of  then  President 
George  Bush  and  state  governors.  These  goals  evolved  into  part  of 
President  Bush’s  America  2000  effort  to  support  new  elementary  and 
secondary  school  designs;  further  support  for  NAS  was  then  provided 
through  the  Clinton  administration’s  Goals  2000. 

Despite  this  general  federal  support,  NAS  and  the  development  of  the 
designs  were  initially  funded  solely  by  the  private  sector.  NAS  sought 
to  engage  the  nation’s  best  educators,  business  people,  and  re¬ 
searchers  in  the  task  of  creating,  testing,  and  fostering  the  implemen¬ 
tation  of  schoolwide  designs  not  constrained  by  existing  reg^ations, 
work  rules,  and  conventions.  Thus,  NAS’s  aim  was  to  help  develop 
“break  the  mold”  schools  to  provide  innovative  options  within  the 
public  education  sector  as  a  way  to  better  educate  students  for  the 
societal  needs  of  the  21st  century. 

The  initial  goed  of  NAS  was  to  help  a  large  number  of  schools  change 
their  organization  and  practices  so  as  to  improve  student  learning. 
To  make  this  goal  a  reality,  NAS  organized  its  work  into  sever^ 
phases:  1 

•  A  competition  phase  to  solicit  proposals  and  select  designs; 

•  A  development  phase  of  one  year  to  develop  the  ideas  in  the  pro¬ 
posals  in  concrete  ways; 


^A  more  detailed  description  of  the  history  of  the  NAS  initiative  and  the  design  teams 
appears  in  Bodilly,  1998;  Gleiman,  1998;  and  Stringfield,  Ross,  and  Smith,  1996.  See 
also  Herman,  1999;  Ball  et  al.,  1998;  Stringtield  and  Datnow,  1998;  Datnow 
and  Stringfield,  1997;  Ross  et  al.,  1997,  1998;  and  Stringfield,  Millsap,  and  Herman, 
1997.  For  descriptions  of  NAS  and  the  design  teams  on  the  Web,  see 
http://www.naschools.org,  which  has  links  to  each  design  team’s  Web  site. 
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•  A  demonstration  phase  of  two  years  to  pilot  the  designs  in  real 
school  settings;  and 

•  A  scale-up  phase  in  which  the  designs  would  be  widely  diffused 
in  partnering  jurisdictions  across  the  nation. 

RAND  first  helped  NAS  to  design  a  request  for  proposals  (RFP)  and 
conduct  a  national  competition  for  design  proposals.  Subsequently, 
RAND  studied  the  initial  implementation  of  the  designs  during 
the  development  and  demonstration  phases,  providing  feedback  to 
both  NAS  and  the  design  teams  (Bodilly  et  al.,  1995;  Bodilly,  1996; 
Mitchell,  1996).  Figure  1.1  shows  the  NAS  phases  and  RAND’s  roles. 

KEY  QUESTIONS  UNDERLYING  RAND’S  ASSESSMENT 

With  the  beginning  of  scale-up  of  the  NAS  designs  in  the  fall  of  1995, 
RAND’s  research  agenda  expanded.  We  began  to  address  broader 
questions  to  document  the  progress  of  the  NAS  initiative: 

•  What  were  student  performance  and  school  climate  like  in  the 
NAS  schools  before  Aey  implemented  the  designs? 
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Figiure  1.1 — Phases  of  New  American  Schools  Initiative  and  RAND  Roles 
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•  How  have  the  designs  and  the  assistance  they  provide  evolved 
over  time? 

•  Are  the  critical  components  of  the  NAS  designs  being  imple¬ 
mented  across  a  wide  array  of  schools?  Why  or  why  not? 

•  Do  the  NAS  designs  extend  beyond  changes  in  school  organiza¬ 
tion  and  governance  and  permeate  classrooms  to  change  cur¬ 
riculum  and  instruction? 

•  Over  time,  what  is  the  progress  of  the  schools  being  assisted  by 
NAS  design  teams  in  improving  student  and  school  perfor¬ 
mance? 

This  report  lays  out  RAND’s  analytic  plan  for  addressing  these  ques¬ 
tions.  It  also  provides  an  answer  to  the  first  question:  V^at  were  the 
NAS  schools  like  before  they  implemented  the  designs?  For  this  re¬ 
port,  we  rely  on  data  from  partnering  school  districts  and  retrospec¬ 
tive  accounts  provided  by  principals  during  phone  interviews  con¬ 
ducted  in  the  spring  of  1997.  These  data  provide  information  about 
the  schools’  demographic,  climate,  and  performance  characteristics 
at  the  beginning  of  NAS’s  scale-up  phase. 

ORGANIZATION  OF  THE  REPORT 

Chapter  Two  describes  NAS’s  approach  to  schoolwide  reform  and 
RAND’s  overall  analysis  plan  and  data  collection  efforts  for  assessing 
the  NAS  initiative  in  terms  of  the  five  key  questions  listed  above.  We 
discuss  the  importance  of  a  more  thorough  empirical  understanding 
of  schoolwide  reforms  or  comprehensive  school  reform  as  the  nation 
continues  to  grapple  with  how  to  improve  the  learning  opportunities 
of  its  students.  Chapter  Three  provides  details  of  the  RAND  sample 
and  presents  a  baseline  description  of  the  NAS  schools  in  our  sample 
in  terms  of  their  demographic  and  school  climate  characteristics.  We 
also  summarize  school  achievement  levels  during  the  baseline  year 
to  provide  a  better  understanding  of  what  the  NAS  schools  were  like 
before  the  design  teams  began  assisting  them.  A  brief  summary  of 
our  findings  and  the  work  ahead  appears  in  Chapter  Four. 


Chapter  Two 


TOWARD  AN  UNDERSTANDING  OF  SCHOOLWIDE 
REFORMS:  ASSESSMENT  OF  THE  SCALE-UP  PHASE 


The  goal  of  NAS  is  to  improve  student  learning.  To  make  this  goal  a 
reality,  NAS  organized  its  work  into  several  phases:  a  call  for  “break 
the  mold”  proposals  for  school  designs,  development  and  pilot  im¬ 
plementation  of  those  proposals,  and  a  diffusing  of  the  developed 
designs  to  a  large  number  of  schools  within  partnering  school  dis¬ 
tricts  (NAS’s  scale-up  phase).  (For  more  information  about  the  evo¬ 
lution  of  NAS,  see  Bodilly  et  al.,  1995;  Bodilly,  1998;  Glennan,  1998.) 

While  private  sponsorship  of  school  innovation  is  not  new  histori¬ 
cally  (Tyack  and  Cuban,  1995),  the  NAS  initiative  is  significant  be¬ 
cause  of  its  focus  on  establishing  partnerships  among  school  designs, 
districts,  and  schools.  NAS  views  such  partnerships  as  key  to  helping 
a  large  number  of  schools  change  their  organization  and  practices  so 
as  to  improve  student  achievement. 

KEY  ELEMENTS  OF  THE  DESIGNS 

In  September  1995,  NAS  and  its  partners  began  to  bring  the  designs 
to  scale— that  is,  to  implement  designs  more  widely  within  partner¬ 
ing  districts.  As  NAS  entered  the  scale-up  phase,  there  were  seven 
design  teams;^ 


1  Another  design,  Urban  Learning  Centers  (ULC),  was  implementing  in  the  Los  Angeles 
area.  This  design  team  was  not  included  in  the  NAS  portfolio  when  scale-up  beg^ 
because  it  had  not  shown  the  capacity  to  go  to  scale.  It  since  has  shown  this  capacity 
and  is  now  being  marketed  by  NAS  as  one  of  its  designs. 
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•  Audrey  Cohen  College  (AC)  (recently  renamed  Purpose-Centered 
Education); 

•  Authentic  Teaching,  Learning,  and  Assessment  for  All  Students 
(ATLAS); 

•  Co-NECT  Schools  (CON); 

•  Expeditionary  Learning  Outward  Bound  (ELOB); 

•  Modern  Red  Schoolhouse  (MR); 

•  National  Alliance  for  Restructuring  Education  (NARE)  (recently 
renamed  America’s  Choice  Design  Network);  and 

•  Roots  and  Wings  (RW). 

Each  design  has  unique  features,^  yet  all  tend  to  emphasize  school 
change  in  the  following  areas: 

•  Organization  and  governance; 

•  Professional  life  of  teachers; 

•  Content  and  performance  expectations; 

•  Curriculum  and  instructional  strategies;  and 

•  Parent  and  community  involvement. 

The  area  of  organization  and  governance  refers  to  the  authority  rela¬ 
tions  among  the  various  parties  in  the  school.  An  example  of  a 
change  in  a  governance  arrangement  is  a  reorganization  of  the  deci¬ 
sionmaking  processes  for  budgets  and  staffing  to  include  teachers 
and  other  school  employees  and  parents.  Giving  authority  to  the 
school  site  has  received  a  great  deal  of  attention  in  the  education 
community.  According  to  Murphy  (1991),  the  central  focus  on  gov¬ 
ernance  restructuring  stems  from  a  belief  that  change  must  reside 
with  those  who  are  closest  to  the  learners  (see  also  Bryk  et  al.,  1998). 
NAS  and  many  of  the  designs  strongly  share  this  belief. 

The  professional  life  of  teachers  refers  to  the  roles  that  teachers  play 
and  the  relationships  in  which  they  participate  during  the  school 


Currently,  RAND  is  documenting  the  continued  development  and  adaptation  of  the 
designs  over  time — from  the  initial  development  of  the  designs’  educational  ideas 
during  NAS's  RFP  phase,  through  the  phases  in  which  these  ideas  met  the  economic, 
sociological,  and  political  realities  of  implementation  in  schools. 
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day.  In  effect,  when  referring  to  restructuring  schools,  particularly 
those  in  poor,  urban  areas,  what  is  involved  here  is  overhauling  the 
conditions  under  which  teachers  work  by  changing  their  responsibil¬ 
ities  and  tasks  and  by  developing  a  more  professional  culture  in 
schools  (Newmann  and  Associates,  1996;  Murphy,  1991;  Sykes,  1990; 
Wise,  1989).  In  contrast  to  an  environment  in  which  teachers  work  in 
isolation  without  contact  with  their  colleagues  (Lortie,  1970),  design 
teams  aim  to  build  a  collaborative  environment  for  teachers.  Thus,  it 
is  important  to  understand  the  extent  to  which  teachers  collaborate, 
engaging  together  in  activities  such  as  professional  development, 
common  planning  time,  and  critiquing  each  other’s  instruction. 
Moreover,  it  is  important  to  understand  the  professional  develop¬ 
ment  activities  of  teachers  and  how  these  relate  to  changes  in  class¬ 
room  activities. 

Each  of  the  designs  aims  to  bring  all  students  to  high  standards,  even 
though  each  may  use  different  means  to  attain  this  goal.  To  monitor 
whether  designs  are  making  progress  toward  this  end,  critical  indica¬ 
tors  might  include  the  degree  to  which  student  assessments  are  ex¬ 
plicitly  linked  to  academic  standards,  teachers  make  performance 
expectations  explicit  to  students,  and  the  curriculum  and  perfor¬ 
mance  standards  are  consistent  and  coherent  across  grade  levels. 

Most  of  the  designs  are  concerned  with  shaping  student  experiences 
within  classrooms  to  further  academic  achievement  growth.  NAS 
designs  embrace  alternative  instructional  strategies  that  involve  dif¬ 
ferent  relationships  between  teachers  and  students  and  between 
students  and  subject  matter.  Yet,  again,  each  design  differs  some¬ 
what  in  the  specific  nature  of  these  activities.  Conventional  class¬ 
rooms  are  often  characterized  as  “teachers  talk  at  students  and  fill 
their  heads  with  knowledge,”  and  “students  respond  with  the  correct 
answers  at  appropriate  times”  (see  Gamoran  et  al.,  1995;  Sizer,  1984; 
Powell,  Farrar,  and  Cohen,  1985).  In  contrast,  design  teams  tend  to 
emphasize  alternative  instructional  practices,  such  as  students  work¬ 
ing  in  small  groups,  using  manipulatives,  engaging  in  student-led 
discussions,  and  working  on  group  and/or  individual  projects  that 
span  a  long  period  of  time  (e.g.,  a  marking  period  or  semester). 

Closely  related  to  instructional  strategies  are  school  decisions  about 
how  students  are  grouped  for  instruction.  The  effects  of  ability 
grouping  and  “tracking”  on  student  achievement  are  strongly  de- 
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bated  among  educators  and  researchers  (see  Slavin,  1987,  1990; 
Gamoran  and  Berends,  1987;  Oakes,  Gamoran,  and  Page,  1992; 
Hallinan,  1994;  Oakes,  1994).  Yet  most  agree  that  alternatives  to  in¬ 
flexible  grouping  arrangements  are  worth  further  exploration.  Thus, 
the  NAS  designs  have  experimented  with  such  alternative  student 
groupings.  For  example,  students  within  an  ELOB  or  CON  design 
may  have  the  same  teacher  for  several  years.  RW  emphasizes  flexi¬ 
bility  by  grouping  students  by  achievement  level  in  reading  for  part 
of  the  day  and  mixing  students  of  various  achievement  levels  for 
other  subjects.  Students  are  assessed  every  eight  weeks  or  so  to  see  if 
they  would  be  better  served  by  being  placed  in  a  different  group.  In 
short,  each  of  the  designs  is  sensitive  to  the  issue  of  ability  grouping 
and  is  working  with  schools  to  group  students  in  more-effective 
ways. 

Conventional  wisdom  suggests  that  the  parent-child  relationship 
and  parent  involvement  in  the  child’s  education  are  critical  compo¬ 
nents  of  school  success.  The  NAS  designs  have  embraced  this  issue 
as  well.  Several  of  them  (e.g.,  ATLAS  and  RW)  aim  to  have  individuals 
or  teams  within  the  schools  that  serve  as  resources  to  students  and 
families  to  help  integrate  the  provision  of  social  services.  Others 
(e.g.,  AC,  ELOB,  and  NARE)  emphasize  the  need  for  students  to  apply 
their  learning  in  ways  that  directly  benefit  the  community.  Each  de¬ 
sign  desires  that  parents  and  community  members  be  involved  in 
positive  ways  in  the  educational  program. 

NEW  AMERICAN  SCHOOLS  APPROACH 
Design-Based  Assistance 

Because  NAS  believes  that  most  schools  have  benefited  from  fo¬ 
cused,  strategic  assistance  in  their  efforts  to  implement  a  design,  it 
now  considers  the  support  and  development  of  design-based  assis¬ 
tance  organizations  to  be  its  most  important  accomplishment.^ 


^To  a  certain  extent,  this  approach  to  supporting  the  diffusion  of  designs  and  design- 
based  assistance  can  be  characterized  as  NAS’s  "theory  of  action”  (Weiss,  1972, 1995, 
1997).  Weiss's  conceptualization  grounds  educational  research  and  evaluation  in 
program  theories  of  change  to  articulate  the  explicit  or  implicit  theories  about  how 
and  why  programs  will  work.  Weiss  (1995,  p.  69)  states  four  major  reasons  why  it  is 
useful  to  ground  educational  research  and  evaluation  of  educational  programs  on  the 
theories  of  change  underlying  educational  programs:  (1)  such  research  concentrates 
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NAS’s  approach  to  schoolwide  reform  relies  on  such  design-based 
assistance  “as  its  cornerstone”  (Glennan,  1998,  p.  14).  In  fact,  NAS’s 
current  mission  is  to  improve  student  achievement  for  large  num¬ 
bers  of  students  through  design-based  assistance  (New  American 
Schools  Development  Corporation,  1999). 

A  critical  aspect  of  design-based  assistance  is  the  designs’  commit¬ 
ment  to  providing  ongoing  assistance  to  support  services  aimed  at 
furthering  implementation  and  school  transformation — organiza¬ 
tion,  curriculum,  instruction,  and  professional  development  of  staff. 
That  is,  when  guided  by  a  design  and  assisted  by  an  external  agent 
(the  design  team),  schoolwide  reform  will  promote  greater  program 
cohesiveness  and  coherence  within  the  school,  as  well  as  collabora¬ 
tion  among  the  school’s  staff.  Such  an  approach  will  lead  to  school- 
based  programs  that  better  serve  the  needs  of  students  and  result  in 
improved  academic  outcomes. 

Toward  this  end,  teachers  and  other  staff  in  schools  need  to  have  a 
significant  amount  of  choice  when  adopting  design-based  assis¬ 
tance.  This  is  consistent  with  NAS’s  conviction  that  educational 
change  cannot  be  mandated  “from  above.”  Because  of  this  belief, 
the  designs  require  a  significant  majority  buy-in  on  the  part  of  school 
staff  (e.g.,  75  to  80  percent)  before  implementation  can  begin. 

Engaging  District  Support 

Initially,  the  NAS  strategy  focused  primarily  on  schools.  NAS  be¬ 
lieved  that  a  national  reform  effort  would  be  established  if  schools 
were  transformed  with  the  help  of  designs — one  at  a  time.  But  this 
NAS  strategy  has  evolved: 

Over  the  past  four  years,  we  have  recognized  that  the  power  of 
school-by-school  improvement  is  limited.  The  major  evolution  in 
the  New  American  Schools  strategy  has  been  the  growing  convic- 


evaluation  attention  and  resources  on  key  aspects  of  the  program;  (2)  it  facilitates 
aggregation  of  evaluation  results  into  a  broader  base  of  theoretical  and  program 
knowledge;  (3)  it  asks  program  practitioners  to  make  their  assumptions  explicit  and  to 
reach  consensus  with  their  colleagues  about  what  they  are  attempting  to  do  and  why; 
and  (4)  research  and  evaluation  that  address  the  theoretical  assumptions  embedded 
in  programs  may  have  more  influence  on  both  policy  and  popular  opinion  than  do 
those  tiiat  fail  to  address  them. 
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tion  that,  without  changes  in  the  structure,  policies,  and  practices  of 
school  systems,  good  schools  will  remain  the  exception  (New  Amer¬ 
ican  Schools  Development  Corporation,  1997,  p.  5). 

Thus,  rather  than  breaking  the  molds  of  existing  schools— one  at  a 
time — with  fundamentally  different  models  of  organization,  NAS 
aimed  toward  a  “bottom-up,  top-down”  perspective  that  attempts  to 
assist  (1)  districts  to  establish  a  conducive  operating  environment  for 
the  designs,  and  (2)  schools  to  change  their  organization,  profes¬ 
sional  development,  standards,  curriculum,  and  instruction  to  im¬ 
prove  student  learning. 

NAS  believes  that  the  designs  and  the  assistance  they  provide  are 
likely  to  be  widely  successful  only  if  they  operate  in  jurisdictions  that 
provide  supportive  environments.  In  such  environments,  school 
districts  support  design  implementation  in  a  number  of  ways  (see 
Bodilly  and  Berends,  1999).  For  instance,  before  implementation, 
districts  can  encourage  schools  to  assess  their  needs  so  that  they  can 
ask  probing  questions  of  the  design  teams  about  how  those  needs 
can  best  be  met  within  the  context  of  a  schoolwide  design.  Such  a 
process,  among  others,  can  further  the  appropriate  match  of  the 
designs  to  schools  based  on  district  accountability  demands,  design 
capabilities,  and  school  needs.  Moreover,  in  supportive  operating 
environments,  there  is  a  reasonably  intense  focus  on  design  im¬ 
plementation  for  a  limited  number  of  years  (e.g.,  three  years).  There 
are  also  adequate  resources  for  accomplishing  that  implementa¬ 
tion — a  critical  aspect  that  requires  district  leadership  and  stability 
(see  Bodilly,  1998;  Keltner,  1998).  Such  an  approach  assumes  that 
this  district  focus  on  schoolwide  transformation  will  lead  to  district 
policies  that  conform  to  and  reinforce  consistent  and  coherent 
strategies  for  school  change.  These  include  not  only  resource  allo¬ 
cation,  but  policies  related  to  hiring,  curriculum,  instruction, 
standards,  and  assessments  as  well. 

In  1995,  at  the  beginning  of  the  scale-up  phase,  NAS  partnered  with 
ten  jurisdictions:  Cincinnati,  Dade  County,  Kentucky,  Maryland, 
Memphis,  Philadelphia,  Pittsburgh,  San  Antonio,  San  Diego,  and 
Washington  State.^  The  goal  in  each  jurisdiction  was  to  transform  30 


^The  districts  in  Kentucky,  Pittsburgh,  and  San  Diego  were  partners  with  the  NARE 
design  team.  NARE  was  also  active  in  Washington  State. 
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percent  of  the  schools  within  a  five-year  period  using  NAS  or  other 
schoolwide  designs.  While  chosen  arbitrarily,  the  30  percent  goal 
reflected  the  desire  to  ensure  that  design-based  schools  would  form  a 
critical  mass  within  each  jurisdiction. 

RAND’S  ASSESSMENT 

The  NAS  approach  relies  on  school  designs  and  design-based  assis¬ 
tance  and  requires  the  alignment  of  numerous  institutional,  social, 
political,  and  economic  factors  for  successful  implementation,  con¬ 
sistency  and  coherence,  and  ultimate  success.  It  is  both  ambitious 
and  complex.^  As  a  result,  any  assessment  of  the  NAS  initiative  has  to 
be  multidimensional  and  multifaceted  and  must  rely  on  a  variety  of 
data  and  a  wide  array  of  indicators.  Such  an  assessment  is  a  difficult 
and  analytically  complex  undertaking.  Yet  it  is  critical  to  under¬ 
standing  the  progress  of  schools  implementing  schoolwide  reform, 
especially  since  so  little  research  has  been  conducted  in  this  rela¬ 
tively  new  educational  policy  area  (see  Herman,  1999;  Fashola  and 
Slavin,  1998;  Pogrow,  1998). 

To  monitor  the  progress  of  the  NAS  initiative,  RAND  is  addressing 
the  five  key  questions  outlined  in  Chapter  One.  These  focus  on  is¬ 
sues  related  to  the  types  of  schools  NAS  design  teams  are  assisting, 
the  development  of  the  design  teams  over  time,  changes  in  class¬ 
room  environments  in  design-based  schools  and  their  effects  on  stu¬ 
dent  achievement,  and  the  implementation  and  performance  trends 
in  the  NAS  schools. 

Data  Collection  Efforts 

To  address  the  issues  of  concern,  RAND  has  used  district-provided 
data  and  has  undertaken  a  variety  of  data  collection  efforts:  inter¬ 
views,  focus  groups,  case  studies,  surveys,  observations,  gathering  of 
school  and  classroom  artifacts,  and  document  analysis.  In  a  sup¬ 
plementary  study  of  changes  in  classroom  practice  in  an  urban 
school  district,  we  are  administering  a  commercial  test  to  a  sample  of 
fourth  graders  in  NAS  and  non- NAS  schools.  All  of  these  data  inform 


®For  further  discussions  of  this  complexity,  see  Bodilly,  1998;  Glennan,  1998;  and 
Fuhrman  and  Ritter,  1998. 
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our  understanding  of  the  reform  that  NAS  is  attempting  to  accom¬ 
plish  in  districts,  schools,  classrooms,  and  the  academic  lives  of  stu¬ 
dents  across  the  United  States. 

Monitoring  a  Longitudinal  Sample  of  NAS  Schools.  For  a  longitudi¬ 
nal  sample  of  schools,  RAND  is  gathering  data  from  districts,  schools, 
principals,  and  teachers.  We  began  collecting  data  at  the  beginning 
of  NAS’s  scale-up  phase,  and  we  currently  plan  to  continue  these  ef¬ 
forts  through  the  1999-2000  school  year.  A  variety  of  data  will  help 
us  to  monitor  the  trends  in  implementation  and  performance  at  the 
NAS  sites: 

•  Teacher  surveys  administered  to  all  teachers  in  NAS  schools. 
These  surveys  cover  such  topics  as  a  teacher’s  understanding  of 
the  design;  human,  material,  strategic,  and  other  support  avail¬ 
able  for  design  implementation;  levels  of  implementation  of  key 
design  elements;  classroom  activities  in  a  target  class;  attitudes 
and  perceptions  toward  teaching;  professional  development  ex¬ 
periences;  teacher  background  information;  teacher’s  overall 
support  for  the  design;  satisfaction  with  student  progress;  and 
perceived  effects  of  the  design  on  the  teacher’s  professional  life 
and  on  student  achievement  and  engagement. 

•  Principal  phone  interviews.  These  interviews,  which  take  about 
an  hour  to  complete,  ask  principals  a  series  of  closed  and  open- 
ended  questions  similar  to  those  on  the  teacher  survey.  In  addi¬ 
tion,  we  ask  principals  about  the  aids  and  impediments  to  re¬ 
form,  the  alignment  between  jurisdiction-sponsored  tests  and 
teaching  and  learning,  teachers’  professional  development  ex¬ 
periences,  and  school  organizational  characteristics. 

•  District  data  on  school  performance  indicators.  These  include 
mandated  test  scores,  attendance  rates,  promotion  and  dropout 
rates,  and  school  demographic  characteristics. 

•  Site  visits  to  schools  and  school  districts.  The  purpose  of  these  site 
visits  is  to  obtain  district,  school  administrator,  and  teacher  re¬ 
ports  of  the  progress  of  the  NAS  initiative.  One  set  of  site  visits  is 
aimed  particularly  at  schools  and  school  districts  no  longer  im¬ 
plementing  NAS  designs,  the  objective  being  to  understand  what 
led  to  the  decision  to  stop.  Another  set  focuses  on  particularly 
successful  schools  in  order  to  understand  what  factors  lead  to 
success. 
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Monitoring  Changes  in  Classroom  Practices  in  San  Antonio.  In  the 
spring  of  1997,  RAND  began  collecting  data  in  selected  NAS  and  non- 
NAS  schools  in  San  Antonio,  Texas.  This  effort  was  to  span  two 
school  years,  ending  in  the  spring  of  1999. 

For  about  60  elementary  classrooms,  our  task  was  to  gather  a  wide 
variety  of  data  over  two  school  years: 

•  Survey  data  from  teachers.  These  data  include  information  on 
instructional  practices,  implementation  of  the  designs,  profes¬ 
sional  development,  parental  involvement,  judgments  about  the 
effects  of  the  design  and  support,  and  background  characteris¬ 
tics. 

•  Student-level  data.  These  data,  gathered  for  all  students  in  the 
elementary  classrooms,  include 

—  Student  background  (gender,  race-ethnicity,  age,  at-risk 
status,  poverty,  English  proficiency,  and  prior  test  scores). 

—  Test  data  from  the  Texas  Assessment  of  Academic  Skills 
(TAAS),  which  provides  significant  indicators  for  the  state 
and  district  accountability  systems. 

—  Test  data  from  the  Stanford  Achievement  Open  Ended  Read¬ 
ing  Test,  Version  9  (SAT-9),  which  is  a  supplementary  com¬ 
mercial  test  that  asks  students  not  only  to  read  a  passage  and 
answer  questions,  but  to  explain  their  answers  as  well. 

•  Principal  phone  interviews.  These  interviews  gather  the  same 
information  gathered  from  the  larger  longitudinal  sample  of 
schools  (see  above). 

•  More-specific  data  from  classrooms.  For  a  subsample  of  about  15 
teachers  from  the  60  classrooms,  we  are  to  collect  additional  data 
on  instructional  practices,  such  as 

—  Observations  of  each  teacher  several  times  over  the  course  of 
the  school  year. 

—  Illustrative  samples  of  students’  work. 

—  Teacher  interviews  and  logs  about  assignments,  homework, 
projects,  quizzes  and  tests,  and  papers  or  reports  over  the 
course  of  the  year. 
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In  summary,  we  are  gathering  a  wide  array  of  data  from  districts, 
schools,  design  teams,  principals,  teachers,  and  students.  We  believe 
such  a  comprehensive  data  collection  effort  is  needed  to  monitor 
and  assess  the  progress  of  a  reform  effort  as  complex  and  varied  as 
the  NAS  initiative. 

Analytic  Approach  to  Key  Questions 

We  now  return  to  the  key  questions  of  the  RAND  assessment  of  NAS 
and  highlight  our  analytic  approach  to  each  of  them.  Table  2.1  sum¬ 
marizes  the  questions  and  our  data  collection  activities.  (Instru¬ 
ments  and  protocols  for  these  studies  are  available  from  RAND  upon 
request.) 

AAdiat  Were  the  NAS  Schools  Like  Before  They  Implemented  the  De¬ 
signs?  This  question  is  the  focus  of  this  report,  which  is  the  first  in  a 
series  of  reports  based  on  analyses  of  the  longitudinal  sample  RAND 
is  using  to  monitor  trends  in  implementation  and  performance. 
Most  of  the  information  in  this  report  is  based  on  data  gathered  be¬ 
fore  the  NAS  designs  were  implemented.  As  such,  the  report  is  a 
“baseline.”  With  data  provided  by  districts,  we  describe  what  the 
NAS  schools  were  like  (e.g.,  in  terms  of  school  performance,  poverty, 
and  racial-ethnic  composition)  before  the  designs  began  implement¬ 
ing  £md  assisting.  Moreover,  we  have  retrospective  survey  data  from 
principals  to  describe  what  their  school  climates  were  like  early  on  in 
the  scale-up  phase. 

How  Have  the  Designs  and  the  Assistance  They  Provide  Evolved 
over  Time?  Since  1992,  the  designs  and  their  teams  have  changed  for 
many  different  reasons.  We  are  examining  the  evolution  of  the 
designs  based  on  document  review  and  interviews  over  several  years. 
This  analysis  addresses  the  question:  What  changes  have  been  made 
to  the  designs  and  why?  It  will  provide  crucial  information  on  the 
designs,  the  variation  in  foci  among  them,  and  the  changes  they 
made  over  time  as  they  encountered  implementation  realities  in 
schools  facing  significant  challenges  in  terms  of  accountability 
systems,  low  test  scores,  poverty,  diverse  student  populations,  and 
difficult  school  climates. 


Key  Questions  of  RAND  Assessment  and  Data  Collection  Activities 
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schoois  being  assisted  by  NAS  design 
teams  in  improving  student  and 
school  performance? 
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Are  the  Critical  Components  of  the  NAS  Designs  Being  Imple¬ 
mented  Across  a  Wide  Array  of  Schools?  Why  or  Why  Not?  As 
noted,  we  have  collected  two  years’  worth  of  data  on  teacher- 
reported  design  implementation  levels  and  on  principals’  and  teach¬ 
ers’  views  of  this  effort.  These  quantitative  data  will  be  comple¬ 
mented  by  interviews  with  district  personnel  to  better  understand 
how  changes  in  district  policies  are  related  to  and  affect  design  im¬ 
plementation  in  schools. 

To  address  this  question,  we  wUl  analyze 

•  Changes  in  implementation  indicators,  support  for  the  designs, 
and  teachers’  judgments  about  the  effects  of  the  designs  in  their 
schools.  We  \^1  use  our  teacher  and  principal  survey  data  from 
the  1996-1997  and  1997-1998  school  years  and  information  pro¬ 
vided  through  interviews  of  key  decisionmakers  in  the  different 
jurisdictions.  Our  indicators  measure  implementation  in  various 
areas:  organization  and  governance,  teacher  professional  life,  in¬ 
structional  strategies  and  performance  expectations,  and  in¬ 
volvement  of  parents  and  community. 

•  Insights  from  the  jurisdiction  interviews.  Our  objective  is  to  un¬ 
derstand  the  decision  in  some  jurisdictions  to  stop  implementing 
the  designs  and  the  importance  to  this  decision  of  factors  such  as 
level  and  type  of  district  support,  new  mandates,  and  teacher 
support. 

•  Differences  among  teachers,  both  within  and  among  schools, 
related  to  variations  in  design  implementation.  The  goal  is  to 
know  more  about  how  the  implementation  of  a  NAS  design  be¬ 
comes  a  schoolwide  reform  effort.  As  designs  continue  to  pro¬ 
vide  assistance  and  as  teachers  continue  to  become  more  famil¬ 
iar  with  the  design  team  activities  in  their  schools,  we  expect  in¬ 
creases  in  implementation  levels  and  agreement  among  teachers 
within  schools. 

Do  the  NAS  Designs  Extend  Beyond  Changes  in  School  Organiza¬ 
tion  and  Governance  and  Permeate  Classrooms  to  Change  Cmricu- 
lum  and  Instruction?  A  principal  hypothesis  of  the  NAS  initiative  is 
that  schoolwide  reform  will  be  successful  when  design  teams  provide 
assistance  to  schools  to  fully  implement  the  key  design  components. 
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A  critical  aim  of  this  schoolwide  reform  is  to  improve  classroom  in¬ 
struction  and  thereby  improve  student  achievement  and  engage¬ 
ment. 

This  question  focuses  on  changes  in  classroom  practices.  To  answer 
it,  RAND  is  conducting  a  supplementary  study  of  classrooms  in  San 
Antonio  to  better  understand  the  differences  in  instructional  prac¬ 
tices  between  NAS  and  non-NAS  classrooms. 

As  part  of  this  study,  we  are  gathering  a  rich  array  of  classroom-level 
data  through  observations,  teacher  interviews  and  focus  groups, 
classroom  artifacts,  and  teacher  surveys.  Such  information  will  help 
provide  answers  to  the  question.  Do  NAS  teachers  and  students  in¬ 
teract  with  each  other  and  with  subject  materials  in  ways  that  reflect 
the  design  teams’  curricular  and  instructional  theories  of  change? 
Our  analyses  and  reporting  of  these  data  will  include  descriptive  re¬ 
sults  about  whether  instructional  practices  promoted  by  the  designs 
are  occurring  in  NAS  classrooms  and  will  compare  these  practices  to 
those  in  non-NAS  classrooms. 

Whether  design-promoted  instructional  practices  are  related  to  stu¬ 
dent  learning  is  clearly  an  important  issue,  and  the  data  we  collect  in 
this  effort  will  help  us  to  address  it.  We  describe  the  analyses  related 
to  the  relationships  among  design  implementation,  instructional 
practices,  and  student  achievement  below. 

Over  Time,  What  Is  the  Progress  of  the  Schools  Being  Assisted  by 
NAS  Design  Teams  in  Improving  Student  and  School  Performance? 
Because  the  main  goal  of  NAS  and  the  design  teams  is  to  improve 
student  learning,  this  is  a  crucial  question  in  our  assessment  of  the 
NAS  initiative.  However,  as  stated  earlier,  a  variety  of  institutional, 
social,  political,  and  economic  factors  must  be  aligned  if  implemen¬ 
tation  is  to  be  successful  and  school  performance  improved.  To  ad¬ 
dress  this  question,  our  assessment  draws  on  a  variety  of  data,  indi¬ 
cators,  and  analyses.  Melding  these  together  is  a  difficult  and  analyt¬ 
ically  complex  undertaking.  Even  so,  understanding  the  progress  of 
schools  implementing  schoolwide  reform  is  important,  particularly 
in  today’s  educational  policy  environment. 

Our  analyses  informing  this  question  include  the  following: 
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•  For  eight  jurisdictions,  we  will  analyze  changes  in  performance 
outcomes  between  1994-1995  (base  year)  and  1997-1998.  That 
is,  we  will  examine  the  progress  of  NAS  schools  in  our  sample 
over  three  years  of  implementation  during  the  scale-up  phase. 

•  We  will  use  a  wide  array  of  data  to  help  explain  these  implemen¬ 
tation  and  performance  trends.  For  example,  we  will  link  im¬ 
plementation  measures  to  school  performance  indicators.  In 
addition,  interview  and  focus  group  information  from  case  stud¬ 
ies  will  provide  a  rich  description  of  what  has  happened  in  these 
schools.  That  is,  not  only  will  we  monitor  the  performance 
trends  in  a  large  sample  of  NAS  schools,  we  will  also  provide  ex¬ 
planations  for  why  those  trends  occurred. 

•  We  will  use  the  San  Antonio  classroom  data  to  examine  the  rela¬ 
tionships  among  implementation,  classroom  instruction,  and 
student  achievement.  We  will  examine  the  relationships  be¬ 
tween  student  achievement  and  a  variety  of  individual  and  social 
factors,  including  (1)  student  characteristics  (gender,  race- 
ethnicity,  at-risk  status,  age,  English  proficiency);  (2)  students’ 
prior  achievement  (i.e.,  third-grade  mathematics  and  reading 
scores);  (3)  classroom  instruction;  (4)  design  team  activities;  and 
(5)  teacher  background  characteristics  (education  level,  experi¬ 
ence,  age,  gender,  and  race-ethnicity).  To  more  accurately  cap¬ 
ture  changes  in  student  performance,  we  will  have  to  assess  the 
changes  in  student  composition  problems,  as  well  as  the  mobil¬ 
ity  of  teachers  in  our  sample. 

•  As  part  of  the  San  Antonio  study,  we  will  track  the  test  scores  of  a 
fourth-grade  cohort.  We  will  be  able  to  obtain  scores  for  the 
fourth  graders  in  our  spring  1998  sample,  as  well  as  their  third- 
grade  (1997)  and  fifth-grade  (1999)  scores.  So,  over  the  course  of 
this  study,  we  will  be  able  track  scores  for  this  year’s  cohort  from 
grade  3  through  grade  5.  We  will  compare  the  trends  of  these 
NAS  students  to  the  trends  of  their  non-NAS  counterparts. 

•  Because  our  previous  work  indicates  that  school-level  differ¬ 
ences  contribute  significantly  to  the  observed  variation  in  imple¬ 
mentation  (Bodilly,  1998),  we  intend  in  the  spring  of  2000  to 
explore  this  issue  further  by  visiting  schools  that  claim  to  be 
implementing  the  designs.  On  the  basis  of  previous  analyses  of 
the  longitudinal  school  sample  described  above,  we  will  classify 
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implementing  schools  as  having  produced  low  or  high  per¬ 
formance  outcomes.  Differences  among  these  two  groups  will 
be  explored  to  provide  additional  information  about  why  per¬ 
formance  differs  in  schools  implementing  NAS  designs. 

Our  analyses  of  student  performance  would  benefit  greatly  ft'om  the 
inclusion  of  individual-level  test  score  data  directly  comparable 
across  all  implementing  sites.  However,  participating  districts  were 
unwilling  to  consider  additional  testing  because  of  the  burden  on 
schools,  teachers,  and  students.  Moreover,  the  cost  of  such  testing 
could  not  be  accommodated  within  the  available  funding.  Thus, 
only  in  the  study  of  classroom  instruction  is  it  feasible  to  administer 
a  supplemental  commercial  test.  While  specialized  tests,  better 
matched  to  the  design  objectives,  might  have  provided  relevant  in¬ 
formation  on  the  effects  of  the  designs,  the  public  expects  perfor¬ 
mance  to  be  measured  against  local  accountability  measures,  which 
are  what  our  analyses  are  based  on. 

We  fully  realize  the  problems  inherent  in  attempting  to  identify  a 
“design  team”  effect  independent  of  the  efforts  of  other  curricular 
and  instructional  programs  being  implemented  in  NAS  schools. 
Nonetheless,  the  combined  analyses  will  provide  important  insights 
into  levels  of  implementation,  extent  of  assistance  as  reported  by 
teachers,  professional  development,  and  the  usefulness  of  the  de¬ 
signs  and  design-based  assistance  for  creating  positive  classroom 
and  school  environments  that  improve  student  learning. 

SIGNIFICANT  FEATURES  OF  RAND  RESEARCH 

The  RAND  assessment  outlined  above  has  a  number  of  significant 
qualities  designed  to  capture  the  complex  nature  of  the  NAS  reform: 

•  It  keeps  a  constant  focus  on  implementation,  using  various 
methods  to  determine  whether  designs  can  be  implemented  in 
real  schools  facing  significant  challenges  and  whether  the  pro¬ 
cess  for  helping  schools  to  choose  among  designs  is  effective. 

•  It  recognizes  that  school  transformation  and  performance  are 
the  joint  products  of  a  number  of  important  factors:  the  design 
itself,  the  assistance  provided  by  the  design  team,  the  environ¬ 
ment  for  implementation  provided  by  the  school  district,  and  the 
readiness  of  the  school.  While  it  cannot  possibly  isolate  these 
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factors  completely  from  each  other,  the  evaluation  tries  to  pro¬ 
vide  some  understanding  of  the  contribution  of  each. 

•  Its  design  emphasizes  school  improvement  over  time.  Rarely 
have  data  been  gathered  on  a  similar  set  of  schools  over  time  to 
monitor  the  progress  of  implementation  and  other  crucial  orga¬ 
nizational  and  performance  factors.  Within  RAND’s  longitudinal 
sample  of  NAS  sites,  schools  serve  as  their  own  “controls.”  Over 
time,  we  examine  whether  each  school’s  test  scores  improve,  and 
we  compare  changes  in  school  test  scores  to  changes  in  district 
test  scores.  The  reason  we  have  not  chosen  to  try  more-complex 
quasi-experimental  evaluation  designs  is  that  they  most  likely 
could  not  be  effectively  implemented,  because  of  NAS’s  overall 
scale-up  strategy. 

•  Its  design  reflects  an  understanding  that  school-centered  reform 
requires  substantial  time.  Our  assessment  spans  a  five-year  pe¬ 
riod. 

In  summary,  RAND’s  research  on  NAS  strives  to  be  a  comprehensive 
assessment  that  will  not  only  inform  NAS,  the  design  teams,  and  the 
partnering  jurisdictions,  but  will  also  provide  important  lessons  for 
those  interested  in  comprehensive  school  reform. 


_ Chapter  Three 

BASELINE  DESCRIPTION  OF 
NEW  AMERICAN  SCHOOLS 


In  this  chapter,  we  provide  a  baseline  description  for  the  NAS  schools 
in  our  sample  by  addressing  the  following  question:  What  were  the 
demographic,  climate,  and  performance  characteristics  of  the 
schools  that  design  teams  began  assisting  at  the  beginning  of  scale- 
up?  The  unit  of  analysis  here  is  the  school;  our  aim  is  to  understand 
what  the  characteristics  of  NAS  schools  were  before  the  designs  part¬ 
nered  with  them. 

We  first  discuss  the  sample  of  schools  used  for  the  descriptive  results. 
We  then  describe  the  NAS  schools  in  terms  of  their  demographic 
characteristics — i.e.,  proportion  of  school  in  poverty  and  proportion 
of  students  from  racial-ethnic  minority  backgrounds.  We  also  dis¬ 
cuss  these  schools’  climate  characteristics  and  compare  them  to  a 
national  sample.  Finally,  we  summarize  the  schools’  performance 
levels  before  the  designs  began  implementation  by  presenting  the 
baseline  performance  data  provided  by  district  data  records. 


SAMPLE  OF  NEW  AMERICAN  SCHOOLS 

The  sample  consists  of  those  schools  initiating  implementation  of 
NAS  designs  in  either  the  1995-1996  or  the  1996-1997  school  year  in 
eight  jurisdictions:^ 


'^When  we  decided  on  the  longitudinal  sample  of  schools,  Maryland  and  San  Diego 
were  not  far  enough  along  in  their  implementation  to  warrant  inclusion  in  our 


21 
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•  Cincinnati 

•  Dade  County 

•  Kentucky 

•  Memphis 

•  Philadelphia 

•  Pittsburgh 

•  San  Antonio 

•  Washington  State 

Our  choice  of  jurisdictions  reflected  both  our  desire  to  obtain  a 
sample  that  included  all  the  designs  participating  in  the  scale-up 
phase  and  our  judgment  that  the  costs  of  working  in  the  additional 
jurisdictions  would  not  yield  commensurate  benefits.  While  juris¬ 
dictions  and  their  support  of  the  NAS  reform  will  no  doubt  continue 
to  change  over  time,  the  jurisdictions  we  chose  reflect  a  range  of 
support  for  implementation — from  a  great  deal  to  very  little  (see 
Bodilly,  1998). 

Our  aim  was  to  have  a  “census”  sample  of  NAS  schools  within  the  ju¬ 
risdictions  selected.2  That  is,  we  wanted  to  gather  information  from 
all  the  principals  and  teachers  at  these  sites.  Based  on  our  conversa¬ 
tions  with  NAS  in  the  late  summer  and  early  fall  of  1996,  we  found 
that  there  were  256  schools  implementing  NAS  designs  across  these 
eight  jurisdictions.  However,  after  further  conversations  with  design 
teams,  jurisdictions,  and  the  schools,  the  sample  was  reduced  to  184 
for  several  reasons: 


planned  data  collection  efforts.  Since  then,  several  of  the  design  teams  report  that 
they  are  implementing  in  Maryland  and  San  Diego. 

^It  is  important  to  note  that  our  sample  of  NAS  sites  is  drawn  from  a  set  of  NAS  schools 
that  expressed  Interest  in  implementing  designs  in  districts  that  had  formed  a 
partnership  with  NAS.  While  this  may  imply  some  bias,  it  is  difficult  to  know  which 
way  the  bias  goes  in  all  instances,  because  both  the  degree  of  free  choice  that  schools 
had  and  the  degree  to  which  jurisdictions  supported  the  NAS  initiative  varied  (see 
Bodilly,  1998;  Bodilly  and  Berends,  1999). 
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•  Fifty-one  RW  schools  in  Dade  County  were  low-performing  and 
on  the  verge  of  serious  sanctions  by  the  state,  so  the  district 
promised  them  that  they  would  not  be  burdened  with  research. 

•  Another  21  schools  declined  to  participate  because  they  did  not 
want  to  be  burdened  with  research,  were  not  implementing,  or 
had  dropped  the  design. 

Table  3.1  breaks  down  the  184  schools  in  the  longitudinal  sample  by 
jurisdiction. 

SOURCES  OF  DATA 

Data  for  the  analyses  reported  here  are  drawn  from  publicly  reported 
district  data  on  school  demographics  (e.g.,  poverty  and  racial-ethnic 
composition)  and  test  scores.  In  addition,  we  conducted  phone  in¬ 
terviews  with  school  principals  in  the  spring  of  1997  to  gather  retro¬ 
spective  data  on  school  climate  characteristics.  These  interviews 
each  took  about  one  hour  to  complete. 

RESPONSE  RATES 

The  principals  of  170  of  the  184  schools  (92  percent)  responded  to 
our  phone  interviews  in  the  spring  of  1997.  The  attained  sample  of 
170  principal  responses  is  shown  by  district  and  design  in  Table  3.2. 
We  rely  on  these  responses  for  the  descriptions  provided  next. 

Table  3.1 

1995-1996  Target  Sample  for  RAND  Longimdinal  Sample 


Jurisdiction 

AC 

ATLAS 

CON 

ELOB 

MR 

NARE 

RW 

Total 

Cinciimati 

5 

5 

6 

16 

Dade  County 

5 

4 

1 

3 

4 

17 

Kentucky 

51 

51 

Memphis 

5 

5 

5 

5 

4 

9 

33 

Philadelphia 

12 

4 

2 

18 

Pittsburgh 

12 

12 

San  Antonio 

8 

5 

13 

Washington  State 

8 

16 

24 

Total 

10 

25 

18 

19 

14 

79 

19 

184 
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Table  3.2 

Principal  Interview  Sample,  Spring  1997 


Jurisdiction 

AC 

ATLAS 

CON 

ELOB 

MR 

NARE 

RW 

Total 

Cincinnati 

5 

5 

6 

16 

Dade  County 

5 

4 

1 

3 

4 

17 

Kentucky 

44 

44 

Memphis 

5 

5 

5 

5 

4 

9 

33 

Philadelphia 

9 

2 

2 

13 

Pittsburgh 

10 

10 

San  Antonio 

8 

5 

13 

Washington  State 

8 

16 

24 

Total 

10 

22 

16 

19 

14 

70 

19 

170 

DEMOGRAPHIC  CHARACTERISTICS  OF  SCHOOLS 

Most  of  the  schools  receiving  design  team  assistance  could  be  con¬ 
sidered  socially  and  academically  disadvantaged  in  terms  of  poverty, 
racial-ethnic  composition,  climate,  and  student  test  scores.  The  NAS 
sites  in  our  sample  are  below  “average”  when  a  number  of  school 
characteristics  are  compared  to  national  norms. 

About  one-third  of  the  nation’s  students  receive  free  and  reduced- 
price  lunches.3  On  average,  55  percent  of  the  students  in  each  of  the 
NAS  schools  in  our  sample  can  be  categorized  as  being  poor. 

Figure  3.1  reveals  that  most  of  the  design  teams  are  assisting  schools 
in  partnering  jurisdictions  with  high  concentrations  of  poor  stu- 


^We  measure  school  poverty  as  the  percentage  of  students  within  a  school  who  receive 
free  and/or  reduced-price  lunches.  Because  all  students  in  the  San  Antonio  In¬ 
dependent  School  District  receive  free  lunches,  poverty  there  is  measured  as  students 
within  a  school  who  are  eligible  for  free  and/or  reduced-price  lunches.  In 
Philadelphia,  free/reduced-price  lunch  information  is  not  available,  so  the  percentage 
of  students  receiving  Aid  to  Families  with  Dependent  Children  (AFDQ  benefits  is  used 
in  its  place.  Reduced-price  lunches  and  AFDC  are  both  indicators  used  for  allocation 
of  federal  Title  I  funding  under  the  1994  Improving  America’s  Schools  Act  (see 
American  Association  of  School  Administrators,  1995;  http://www.ed.gov/legislation/ 
ESEA).  Currently,  schools  can  use  their  Title  I  funding  for  schoolwide  improvement  if 
at  least  50  percent  of  the  students  within  the  school  are  from  poor  families  (see  U.S. 
Department  of  Education,  1993;  Borman  et  al.,  1996).  The  national  averages  for  both 
school  poverty  and  minority  composition  are  based  on  the  nationally  representative 
Schools  and  Staffing  Survey  (SASS)  of  1993-1994  (for  a  further  description  of  these 
SASS  data,  see  U.S.  Department  of  Education,  1996). 


Figure  3.1 — Percentage  of  Poor  Students  in  NAS  Schools,  1994-1995 

dents.  For  example,  the  NAS  schools  in  Cincinnati,  Dade  County, 
Memphis,  Pittsburgh,  and  San  Antonio  are  serving  mostly  poor 
student  populations — i.e.,  over  two-thirds  of  the  students  receive 
free  or  reduced-price  lunches  at  these  NAS  sites.  The  percentage  of 
poor  in  our  Philadelphia  jurisdiction  is  based  on  a  more  stringent 
measure  of  poverty:  percentage  of  students  receiving  AFDC  benefits. 
Thirty-eight  percent  of  the  students  in  the  Philadelphia  NAS  schools 
are  receiving  AFDC  benefits,  reflecting  the  high  poverty  levels  in  this 
large,  urban  district. 

The  design  team  in  Kentucky,  NARE,  assisted  schools  similar  to  those 
in  the  nation  in  terms  of  the  percentage  of  students  classified  as 
poor.  In  Washington  State,  the  design  teams,  ATLAS  and  NARE,  are 
assisting  schools  that  are  more  affluent  than  the  national  average.  In 
fact,  if  the  more  affluent  schools  in  the  states  of  Kentucky  and 
Washington  are  excluded,  the  school  poverty  composition  of  the  NAS 
sample  increases  to  68  percent. 

We  also  examined  the  racial-ethnic  compositions  of  the  NAS  schools 
and  compared  them  to  the  national  average.  While  having  a  greater 
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proportion  of  minority  students  in  a  school  maybe  advantageous  for 
students  in  some  respects,  a  disproportionate  number  of  minority 
students  may  also  indicate  problems  associated  with  segregation  and 
racial-ethnic  isolation  in  the  nation’s  schools  (see  Orfleld  and  Eaton, 
1996;  Armor,  1995;  Wells,  1995;  Crain  and  Mahard,  1982). 

About  35  percent  of  the  nation’s  students  are  categorized  as  racial- 
ethnic  minorities  (see  Figure  3.2).  In  contrast,  on  average,  57  percent 
of  the  students  in  NAS  schools  are  minorities.  The  NAS  design  teams 
in  Cincinnati,  Dade  County,  Memphis,  and  San  Antonio  are  assisting 
schools  that  have  a  vast  majority  of  minority  students.  By  contrast, 
the  NAS  schools  in  the  states  of  Kentucky  and  Washington  are  mostly 
nonminority  (non-Hispanic  white)  schools.  If  the  Kentucky  and 
Washington  schools  are  removed  from  the  sample,  over  80  percent  of 
the  students  in  NAS  schools  are  minorities.  Thus,  most  of  the  NAS 
designs  are  assisting  schools  with  disproportionately  high  percent¬ 
ages  of  racial-ethnic  minority  students. 

In  short,  whether  considering  school  poverty  or  racial-ethnic  com¬ 
position,  most  of  the  NAS  sites  are  not  reflective  of  the  nation.  In 
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Figure  3.2 — Percentage  of  Racial-Ethnic  Minority  Students  in  NAS  Schools, 

1994-1995 
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fact,  these  indicators  suggest  the  NAS  designs  are  assisting  mostly 
urban  schools  facing  significant  challenges  related  to  school  poverty 
and  racial-ethnic  segregation. 

SCHOOL  CLIMATE:  NAS  SCHOOLS  COMPARED  TO  THE 
NATION’S  SCHOOLS 

How  do  the  school  climates  in  the  NAS  sites  compare  to  those  of  U.S. 
schools  generally?  In  our  spring  of  1997  phone  interviews,  we  asked 
principals  several  questions  about  school  climate.  Specifically,  we 
asked  them  to  provide  retrospective  reports  about  the  severity  of 
various  problems  in  their  schools.  The  questions  were  taken  firom  a 
national  survey  so  that  we  could  compare  the  NAS  principal  re¬ 
sponses  to  those  of  a  nationally  representative  sample  of  principals.^ 
We  compared  NAS  principals  to  the  nation’s  principals  along  two 
sets  of  school  problems  related  to 

1.  Diverse  student  populations,  as  indicated  by  principal  reports  of 
the  severity  of  school  problems  connected  with 

•  poverty 

•  difficulties  with  the  English  language 

•  racial  tension 

•  lack  of  school  readiness® 

2.  Engagement,  as  indicated  by 

•  student  absenteeism 

•  teacher  absenteeism 

•  verbal  abuse  of  teachers 

•  student  disrespect  of  teachers 


^We  compared  our  NAS  results  to  the  nationally  representative  SASS  of  1993-1994. 
(For  a  further  description  of  these  SASS  data,  see  U.S.  Department  of  Education,  1996.) 

®For  the  sake  of  parsimony,  we  created  a  composite  for  school  readiness,  a  scale 
including  principals’  reports  of  problems  with  student  apathy,  lack  of  academic 
challenge,  students  coming  unprepared  to  learn,  poor  nutrition,  and  poor  health.  The 
alpha  reliability  of  this  scale  was  .80  in  the  NAS  data  and  .83  in  the  SASS  data. 
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In  the  figures  that  follow,  we  present  the  percentage  of  principals  at 
the  NAS  sites  and  in  the  nation  who  rated  these  problems  as 
"moderate”  or  “serious.”  We  report  our  results  separately  for  ele¬ 
mentary  and  secondary  schools  because  the  severity  of  several  of 
these  indicators  differs  across  these  levels. 

Problems  Related  to  Diverse  Student  Populations 

Compared  to  a  national  sample  of  principals,  a  greater  percentage  of 
NAS  principals  report  problems  related  to  poverty,  difficulties  with 
the  English  language,  racial  tension,  and  school  readiness  (see  Fig¬ 
ures  3.3  and  3.4).  For  example,  almost  two-thirds  of  NAS  elementary 
and  secondary  principals  report  moderate  to  serious  problems  re¬ 
lated  to  the  level  of  student  poverty  in  their  schools.  Less  than  50 
percent  of  the  nation’s  principals  report  such  problems. 

In  both  elementary  and  secondary  schools,  there  is  a  difference  be¬ 
tween  the  NAS  sites  and  the  national  sample  of  about  five  to  eight 


Poverty  Difficulties  with  the  Racial  tension  School  readiness 
English  language 


Figure  3.3 — Percentage  of  Elementary  School  Principals 
Reporting  Moderate  to  Serious  Problems  Related  to 
Diverse  Student  Populations 


Baseline  Description  of  New  American  Schools  29 


Poverty  Difficulties  with  the  Racial  tension  School  readiness 
English  language 

Figure  3.4 — Percentage  of  Secondary  School  Principals 
Reporting  Moderate  to  Serious  Problems  Related  to 
Diverse  Student  Populations 

percentage  points  when  considering  difficulties  with  the  English  lan¬ 
guage  and  racicd  tension. 

There  is  a  more  marked  difference  between  NAS  schools  and  the  na¬ 
tion  with  respect  to  school  readiness.  In  fact,  the  severity  of  the 
school  readiness  problems  reported  by  the  NAS  elementary  princi¬ 
pals  is  roughly  twice  that  reported  by  principals  across  the  nation  (17 
vs.  8  percent).  In  secondary  schools,  21  percent  of  the  NAS  principals 
report  moderate  to  serious  problems  with  school  readiness,  com¬ 
pared  to  12  percent  of  the  nation’s  principals. 

Problems  Related  to  Engagement  of  Students  and  Faculty 

The  NAS  schools  also  presented  challenges  related  to  the  engage¬ 
ment  of  students  and  teachers  in  the  education  process.  Compared 
to  the  national  sample  of  principals,  a  greater  percentage  of  NAS 
principals  report  moderate  to  serious  problems  with  student  absen- 
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teeism,  teacher  absenteeism,  verbal  abuse  of  teachers  by  students, 
and  student  disrespect  of  teachers  (see  Figures  3.5  and  3.6).  For  ex¬ 
ample,  over  40  percent  of  NAS  elementary  principals  report  prob¬ 
lems  with  student  absenteeism,  compared  to  20  percent  of  the 
nation’s  principals.  In  addition,  while  11  percent  of  the  nation’s  ele¬ 
mentary  school  principals  report  problems  with  teachers  being 
absent  from  work,  33  percent  of  NAS  principals  report  teacher  absen¬ 
teeism  problems. 

In  secondary  schools,  problems  with  student  verbal  abuse  and  disre¬ 
spect  of  teachers  are  much  greater  in  NAS  schools  than  in  the  nation 
as  a  whole.  For  instance,  about  one-half  of  the  NAS  secondary 
school  principals  report  moderate  to  serious  problems  with  student 
disrespect  of  teachers,  compared  to  about  one-quarter  of  the  na¬ 
tion’s  principals.  Moreover,  the  percentage  of  NAS  secondary  school 
principals  reporting  problems  with  verbal  abuse  of  teachers  is  more 
than  twice  that  of  the  nation’s  principals  reporting  such  problems 
(43  percent  vs.  18  percent). 
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Figiure  3.5 — Percentage  of  Elementary  School  Principals  Reporting 
Moderate  to  Serious  Problems  Related  to  Engagement 
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Figure  3.ft— Percentage  of  Secondary  School  Principals  Reporting 
Moderate  to  Serious  Problems  Related  to  Engagement 


SCHOOL  ACHIEVEMENT  BEFORE  DESIGN 
IMPLEMENTATION 

To  understand  the  progress  of  NAS  schools,  particularly  in  terms  of 
the  schools’  performance  on  achievement  tests,  requires  an  under¬ 
standing  of  schools’  achievement  levels  before  designs  were  imple¬ 
mented. 

In  genertd,  the  NAS  schools  were  in  low-performing,  urban  school 
districts.  Within  these  districts  and  with  few  exceptions,  the  NAS  de¬ 
sign  teams  began  assisting  schools  that  were  scoring  at  or  below  the 
district  average  on  district  or  state-mandated  tests. 

Rather  than  provide  each  comparison  by  grade  level,  jurisdiction, 
and  test,  we  present,  in  Figure  3.7,  a  summary  of  the  comparisons  we 
made  among  grade  levels  and  types  of  tests  in  the  different  districts. 
While  the  tests  and  the  reporting  of  them  to  the  public  differ  dramat¬ 
ically  across  the  jurisdictions,  our  aim  was  to  compare  the  NAS 
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school  scores  to  the  district  averages  and  categorize  them  as  below 
(black  in  Figure  3.7),  at  (gray),  or  above  (white)  those  averages.® 
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Mathematics 

(NAS  sites  compared  to  the  entire 

(NAS  sites  compared  to  the  entire 

district/state) 

district/state) 

NAS  schools  in: 

Elementary  |  Middle  |  High 

Elementary  |  Middle  |  High 

CAT :  California  Achievement  Test 

CTBS:  Comprehensive  Test  of  Basic  Skills 

ITBS:  Iowa  Test  of  Basic  Skills 

KIRIS;  Kentucky  Instructional  Results  Information  System 

SAT-8:  Stanford  Achievement  Test,  Version  8 

SAT-9:  Stanford  Achievement  Test.  Version  9 

TAAS;  Texas  Assessment  of  Academic  Skills 

TAP:  Test  of  Achievement  and  Proficiency 
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Figure  3.7 — Baseline  Achievement  Levels  of  NAS  Sites  Compared  to 
District  or  State  Averages  on  Standardized  Tests 


®We  classified  schools  as  above  or  below  the  district  average  if  they  were  plus  or  minus 
(1)  .2  of  a  standard  deviation  on  the  grade-level  test  scores  and  we  had  access  to 
student-level  test  score  data  (i.e.,  Kentucky),  (2)  5  percentage  points  if  scores  were  re¬ 
ported  as  average  percentile  scores,  or  (3)  7  percentage  points  if  scores  were  reported 
as  median  percentile  scores.  These  cut-points  are  somewhat  arbitrary.  For  example, 
Fashola  and  Slavin  (1998)  suggest  using  somewhat  more  stringent  criteria  (e.g.,  effect 
size  of  .25  of  a  standard  deviation).  Nonetheless,  we  believe  that  the  criteria  above 
suffice  for  the  basic  point  that  even  within  low-performing  districts,  the  NAS  design 
teams  began  assisting  schools  that  were  at  or  below  the  district  average. 
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In  general,  the  NAS  schools  were  scoring  at  or  below  the  district  test 
score  averages  before  the  NAS  designs  began  assisting  them,  even  in 
jurisdictions  that  are  primarily  low-performing  compared  to  state 
averages.  The  NAS  sites  in  Washington  State  are  the  exception 
because  they  tend  to  be  more  affluent,  as  indicated  by  the  per¬ 
centage  of  students  eligible  for  free  and/or  reduced-price  lunches. 


Chapter  Four 

SUMMARY  OF  FINDINGS  AND  WORK  AHEAD 


The  NAS  designs  are  attempting  to  change  the  academic  lives  of  stu¬ 
dents  and  the  professional  lives  of  teachers  in  difficult  environments. 
The  baseline  information  provided  here  shows  clearly  that  the  NAS 
design  teams  began  assisting  some  very  challenged  schools  when 
they  started  implementing  the  designs  during  NAS’s  scale-up  phase. 
For  the  most  part,  NAS  designs  entered  schools  that  were  primarily 
urban,  low-performing,  poor,  and  minority. 

The  NAS  schools  also  have  more  difficult  climates  than  most  of  the 
nation’s  schools  do.  In  comparing  the  NAS  school  climate  indicators 
to  a  national  sample,  we  find  that  NAS  principals  report  greater 
problems  related  to  diverse  student  populations — ^poverty,  difficulty 
with  the  English  language,  racial  tension,  and  school  readiness.  The 
schools  in  which  the  NAS  designs  were  being  implemented  were  also 
facing  challenges  associated  with  engaging  students  and  teachers  (as 
indicated  by  student  and  teacher  absenteeism,  student  verbal  abuse, 
and  student  disrespect  of  teachers). 

The  vast  majority  of  the  NAS  schools  are  in  low-performing,  urban 
districts.  For  the  most  part,  except  in  Washington  State,  the  NAS  de¬ 
signs  entered  schools  that  were  performing  at  or  below  the  district 
average  on  the  mandated  tests  administered  in  each  jurisdiction.  In 
short,  the  NAS  designs  are  not  assisting  a  set  of  socially  or  academi¬ 
cally  advantaged  schools.  Rather,  the  schools  being  assisted  are  fac¬ 
ing  significant  challenges. 

Whether  designs  and  the  assistance  they  provide  to  these  schools 
improve  the  educational  opportunities  and  outcomes  for  the  stu¬ 
dents  is  a  crucial  question.  To  address  this  question,  our  assessment. 
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described  in  this  report,  is  drawing  on  a  variety  of  data,  indicators, 
and  analyses.  Blending  these  together  is  a  difficult  and  analytically 
complex  undertaking.  However,  examining  the  relationships  among 
implementation,  school  and  student  performance,  and  related  social 
and  organizational  factors  is  critical  for  understanding  comprehen¬ 
sive  school  reform  and  its  promise  for  our  nation's  future. 
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